Skip to content

Add OCI image support - #191

Open
henrybear327 wants to merge 23 commits into
sysprog21:mainfrom
henrybear327:oci/setup
Open

Add OCI image support#191
henrybear327 wants to merge 23 commits into
sysprog21:mainfrom
henrybear327:oci/setup

Conversation

@henrybear327

@henrybear327 henrybear327 commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Prior work: #34
Tracking issue: #31

We introduce elfuse-oci, a standalone Go companion binary that owns the OCI image pipeline: pull, unpack, inspect, run, list (alias images), rmi, and prune, backed by a real OCI image-layout store. elfuse itself stays a pure Linux syscall-to-Darwin runtime with no OCI awareness; the two binaries meet only at the existing elfuse --sysroot <rootfs> <entrypoint> <args> launch path.

OCI images are used here strictly as a distribution format for Linux root filesystems — a reproducible replacement for hand-built --sysroot trees. This is not a container runtime. There is no isolation layer: the guest shares the host network identity, PID space, and clock, and unresolved guest paths fall back to host truth. A workload that needs namespaces, cgroups, port mapping, exec into a running container, or a daemon needs a real container runtime, not an ELF personality.

Why a separate Go binary

The OCI ecosystem is a Go ecosystem. Rather than grow the C runtime with an image pipeline, the acquisition/lifecycle half is ~4.4k lines of Go (plus ~7.3k lines of tests) built on github.com/google/go-containerregistry, and validated for on-disk conformance against crane, skopeo, and umoci. Keeping it out of the C tree keeps both sides small and lets each lean on its native tooling.

Design

docs/oci-design.md is the source of truth: the C/Go boundary, the image-layout store and its refs.json pin table, hardened layer application, the run paths, the concurrency/locking model, and an explicit accounting of which OCI features are and are not implemented. docs/usage.md covers the commands; docs/testing.md the offline/CI split; docs/internals.md the host-literal path fallback.

Highlights:

  • Store: spec-shape OCI image-layout other tools can read; refs pinned by digest; an exclusive flock serializes metadata writes so concurrent pulls cannot lose pins.
  • Unpack: os.Root-bounded extraction (no symlink/hardlink escape), correct whiteout/opaque handling, staged temp-dir + atomic rename so readers never see a partial tree.
  • run: resolves Entrypoint/Cmd/Env/User/WorkingDir with env(1)/Docker precedence, guarantees a PATH, resolves symbolic --user against the image's own /etc/passwd+/etc/group (no-follow), injects host /etc/{resolv.conf,hosts,hostname}, then execs elfuse in place so signals and the pid pass through.
  • macOS rootfs: per-digest case-sensitive APFS sparsebundle with per-run clonefile COW; liveness/lifecycle decided by per-digest advisory flocks (attach.lock/run.lock), not pids. --plain-rootfs remains available.
  • Lifecycle GC: rmi/prune use reachability GC (shared blobs survive while any ref reaches them) and never reclaim a cache a live run still holds — the run-lock rides through the exec into elfuse and releases exactly on guest exit.

Try it

make elfuse elfuse-oci

build/elfuse-oci pull alpine:3
build/elfuse-oci inspect alpine:3
build/elfuse-oci list

build/elfuse-oci run alpine:3 /bin/sh -c 'echo hello from elfuse'
build/elfuse-oci run --entrypoint /usr/local/bin/python3 python:3.12 \
  -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'

build/elfuse-oci rmi --force alpine:3
build/elfuse-oci prune --cache --all

Images are stored under $ELFUSE_OCI_STORE, or ~/.local/share/elfuse/oci by default.

CI

Split by runner capability

  • a Linux job exercises the pure-Go store paths (pull, unpack, inspect, lifecycle) plus image-layout conformance and cross-tool interop (crane/skopeo/umoci) with no Hypervisor.framework
  • a hosted macOS job builds and tests the darwin sparsebundle code and a run-less lifecycle smoke

Summary by cubic

Adds OCI image support with a new Go CLI, elfuse-oci, and a shared elfuse_launch so images and positional ELFs use one launch path. Replaces the sidecar with a case‑escape filename codec and a single case‑exact resolver on macOS, and ships a macOS case‑sensitive rootfs with per‑run COW plus a full image lifecycle (pull → run → prune/rmi).

  • New Features

    • Runtime/launch: extracted elfuse_launch; adds --user/--workdir/--env/--clear-env (UID/GID staged so auxv AT_UID/AT_GID match); forwards INT/TERM/QUIT/HUP; uses -- before guest argv; injects host /etc/{resolv.conf,hosts,hostname}.
    • Filenames and paths: new case‑escape codec and a single case‑exact walk replace the sidecar; follows symlink targets in the guest namespace; decodes cwd and getdents names; resolves PT_INTERP via the walk; exec pathnames translate and /proc/self/{exe,fd} reverse‑map; AF_UNIX pathname sockets translate/shorten and reverse‑map in getsockname/recv*; inotify watch paths translate and event names decode; enforces O_NOFOLLOW/O_NONBLOCK for /dev/shm and execveat; unifies create and lookup redirects for /tmp, /var/tmp, and ~/.ccache into the sysroot.
    • OCI lifecycle and run: elfuse-oci (pull, unpack, inspect/list with --json, run, rmi, prune) on github.com/google/go-containerregistry; refs pinned by digest with lock‑serialized updates and reachability GC; --platform (default linux/arm64); run only auto‑pulls when absent, resolves Entrypoint/Cmd/Env/User/WorkingDir, guarantees PATH, resolves symbolic users from in‑image /etc/passwd//etc/group (no‑follow), creates a missing WorkingDir, performs in‑image PATH lookups, and execs elfuse --sysroot so pid/signals pass through.
    • Rootfs caches: macOS case‑sensitive APFS sparsebundles per digest with per‑run clonefile COW and attach/run locks; --plain-rootfs retained with a per‑digest sibling lock that survives exec; rmi/prune skip busy caches, refuse live ones, honor --keep (force discards), and detach mounted bundles when safe.
  • Docs & CI

    • Docs: added docs/filenames.md (filename model), docs/oci-design.md, and updates to docs/usage.md, docs/testing.md, and docs/internals.md.
    • Tests: case‑exact name suite and path‑resolution matrix; exec/cwd regressions; AF_UNIX shortening and reverse‑map; inotify path/name translation; PT_INTERP fallback; tmp‑roots lookup/create consistency; store conformance and interop; darwin sparsebundle round‑trip; end‑to‑end OCI lifecycle and run; per‑image workloads (python/node/go/jvm/c).
    • CI: new hvf-elfuse-setup action and a skip‑list checker; Linux runs OCI store conformance and interop (crane/skopeo/umoci); hosted macOS builds darwin code and a run‑less lifecycle; self‑hosted Apple Silicon runs full HVF flows, execution checks (AF_UNIX, cold/warm boot, dynamic loader), and workloads.

Written for commit 05b184a. Summary will update on new commits.

Review in cubic

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv

This comment was marked as resolved.

@jserv

jserv commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Commands to try out

build/elfuse oci list
build/elfuse oci pull alpine:3
build/elfuse oci inspect alpine:3
build/elfuse oci list
build/elfuse oci rmi --force e7a1a92a5bfe 
build/elfuse oci list
build/elfuse-container run --entrypoint /usr/local/bin/python3 \\n    python:3.12 -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'
build/elfuse oci run alpine:3 /bin/sh -c 'echo hello from elfuse'

Leveraging existing Go-based tools and packages is a great step toward full OCI image support. I agree with this change.

I suggest repositioning the build/elfuse binary as an efficient Linux syscall-to-macOS/Darwin runtime, while implementing build/elfuse-container in Go using OCI-related packages.

In other words, build/elfuse oci should not be considered a valid command. OCI-specific functionality should reside in elfuse-container, not in the elfuse executable.

@henrybear327
henrybear327 force-pushed the oci/setup branch 2 times, most recently from c18f864 to ed17166 Compare July 10, 2026 21:56
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
Runtime file injection de-symlinked only the /etc directory itself; the
per-file os.WriteFile still followed an image-shipped symlink at
etc/{hostname,hosts,resolv.conf}, letting a malicious image redirect
the write outside the rootfs. Named --user resolution had the same
flaw on the read side: a symlinked etc/passwd or etc/group made
lookupPasswd/lookupGroup read host account files.

Route both through os.OpenRoot (the same containment the layer
unpacker already uses): injection unlinks the existing entry and
recreates it O_EXCL, and passwd/group opens are rootfs-bounded.
Regression tests pin both behaviors.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
Mode finalization ran only when an entry carried setuid/setgid/sticky
bits, but the creation modes passed to os.Root.OpenFile and MkdirAll
are masked by the process umask: under e.g. umask 0077 a layer's
0755/0644 entries unpacked as 0700/0600 and were never corrected.

Chmod every created file and directory entry to its exact tar mode
(applyMode), and split the ensure-parent path out (ensureParent) so
finalizing an entry cannot reset the mode of an already-unpacked
parent directory to the 0755 default. Regression test unpacks under
umask 0077 and checks exact modes, including a 0700 parent that a
later child entry must not widen.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
Both run paths infer "already unpacked" from the rootfs path existing
(csrun.go and the plain-directory path in commands.go), but unpackImage
created the destination before applying layers and left it behind on
failure. A run after a failed unpack therefore skipped the unpack and
executed against the truncated tree.

Delete the destination on unpack failure. The cleanup applies only
when unpackImage created the directory itself, so an explicit
pre-existing `unpack --rootfs DIR` target is never removed. Regression
test drives a two-layer image whose second layer fails and checks both
the cleanup and the pre-existing-directory guard.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
pin/rmi updated refs.json with an unlocked load-modify-save cycle and
a fixed temp filename, so two concurrent pulls (or a pull racing an
rmi) could clobber each other's temp file and drop a just-recorded pin
-- a later unpack/run then reports the image as not pulled even though
its pull succeeded. index.json has the same read-modify-write shape
inside the layout package.

Add an exclusive flock on <store>/.lock held across pin's cycle,
addImage's check-append-pin, and rmi's whole resolve-modify-GC
sequence, and give savePins a unique temp name. A 16-writer
concurrent-pin regression test asserts no entry is lost.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
cmdRun treated every s.image failure as "not pulled" and fell into the
auto-pull path, so a corrupt refs.json or unreadable layout triggered a
surprise network pull instead of reporting the store problem.

Introduce an errNotPulled sentinel wrapped by digestFor and
resolvePinnedTarget (user-facing message text is unchanged) and gate
the auto-pull on errors.Is. A regression test pins the two error
kinds apart.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
Two GC robustness fixes:

DirEntry.Info() returning ENOENT (a blob reclaimed by a concurrent
rmi/prune between ReadDir and Info) aborted the whole GC pass; skip
the vanished entry instead, matching the IsNotExist tolerance already
used elsewhere in gc.go.

A last-pin rmi committed the pin removal to refs.json before removing
the manifest descriptor from index.json. If descriptor removal then
failed, the image was stranded: no ref resolves to it, the descriptor
keeps every blob live, and prune never removes descriptors. Reorder
the writes so the same failure window leaves a stale pin over a
removed descriptor, which a retried rmi resolves and completes
(RemoveDescriptors is a filter; re-removal is a no-op). Regression
test forces the descriptor write to fail and checks the pin survives
and the retry finishes.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
pruneCSBundle ran the crash-recovery sweep -- including an
unconditional hdiutil detach -force of any attached volume -- before
checking whether the digest was still pinned, so a non---all
`prune --cache` could rip the rootfs out from under an active run
(the sweep cannot tell a crashed leftover mount from a live one by
mount state alone).

Two guards fix this. The live[key] pin check now runs before the
sweep, so a plain prune never touches a pinned bundle; a crashed
pinned bundle is recovered by the next run's provision or by --all.
And sweepCSBundle now reports a volume busy instead of detaching when
a run-<pid> clone of a live process remains inside it, protecting the
--all and legacy/unpinned paths too.

The gated darwin round-trip now covers the busy path, and folds in two
review fixes of its own: the orphan clone uses a never-assignable pid
(999999999 > kern.maxpid) instead of a reaped pid that the OS could
reuse mid-test, and a t.Cleanup force-detach keeps failed runs from
leaking an attached volume.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
clearDir followed a symlink at the mount-point path, so pre-attach
cleanup of a corrupt or tampered store could empty a directory outside
the OCI cache. Reject a symlinked mount dir with an error instead.

Also drop csMount.imagePath: it was set but never read in production
(every consumer derives <bundle>/rootfs.sparsebundle itself), so the
field was state to maintain with no behavior behind it.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
runMainSubprocess read the stdout pipe to EOF before touching the
stderr pipe -- the sequential-read pattern the os/exec docs warn can
deadlock once the unread stream fills its ~64KB buffer. Hand both
streams to exec.Cmd as bytes.Buffers instead, which the package drains
concurrently; less code and no deadlock potential as outputs grow.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
The jq expression comparing registry truth against the store pin
returned every manifest matching os/arch, so a manifest list with two
matching entries (several variants, or a future extra descriptor)
produced a multi-line string and failed the equality check on a valid
image. Wrap the selection in first(...) and exclude BuildKit
attestation manifests, mirroring what crane.Pull(WithPlatform)
resolves.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
The prune summary presented one number as a uniform on-disk-allocation
figure, but blob bytes come from logical file sizes while cache-dir
bytes come from st_blocks allocation. Say so in the pruneReport doc and
mark the user-facing total approximate rather than pretending to a
single metric.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
v1.18.7 fixes an out-of-bounds read in s2.NewDict. The dependency is
indirect (via go-containerregistry) and nothing here imports the s2
package, so there is no reachable exposure -- this is dependency
hygiene while the report is fresh.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 11, 2026
Every pre-launch failure path unlinks a FUSE-materialized temporary
ELF (elf_host_temp) before returning; the --env/--clear-env OOM branch
was the lone omission and leaked the temp file on disk. Mirror the
sibling paths.

Reported by cubic review on PR sysprog21#191.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cubic-dev-ai[bot]

This comment was marked as resolved.

@henrybear327
henrybear327 marked this pull request as draft July 11, 2026 12:21
@henrybear327

This comment was marked as outdated.

jserv

This comment was marked as resolved.

@henrybear327

This comment was marked as outdated.

@henrybear327
henrybear327 marked this pull request as draft July 20, 2026 00:11
henrybear327 added a commit to henrybear327/elfuse that referenced this pull request Jul 20, 2026
Apply a stored image's layers into a rootfs directory, whiteouts and
all, so `unpack` (and later `run`) can materialize a filesystem tree
from the store without any external tool.

Layer application is hardened against hostile archives: extraction is
bounded by os.Root so no entry, symlink, or hardlink escapes the
destination; parent components are Lstat'd and a symlinked or non-dir
intermediate is replaced with a real directory (containerd/Docker
behavior); opaque whiteouts clear through a real directory only, are
order-independent within a layer, and an invalid bare .wh. entry
fails extraction instead of deleting its parent; a directory entry
replaces a lower-layer non-directory. File bodies are bounded by the
tar header size and short bodies are an error, and permissions are
finalized with an explicit chmod so the host umask cannot skew modes.

setuid/setgid/sticky bits are re-applied where the host allows it, and
degrade gracefully where it does not. An unprivileged chmod that sets
setuid/setgid is rejected with EPERM on macOS when the unpacked file's
inherited group is one the invoking user is not in (a new file takes
its parent directory's group under BSD semantics, e.g. wheel under
/tmp, not the tar's root/shadow owner). Since the rootfs is owned by
the invoking user those bits could not be honored at runtime there
anyway, so they are dropped with a warning naming the lost bit rather
than aborting the whole unpack, which is what lets Debian-family
images and their shadow suite (chage, passwd, ...) unpack at all.
Reported at
sysprog21#191 (comment)

Unpack stages into a temp sibling directory and renames into the
final cache path (keyed by manifest digest under <store>/rootfs/), so
a concurrent reader never observes a partial tree and the loser of a
rename race adopts the winner's complete one. A failed unpack removes
only what it created.

Tests drive whiteouts, opaque ordering, parent-symlink replacement,
hardlink identity, exact-size reads, mode preservation, the special-
bit degrade decision, and the staged-rename semantics over synthetic
layer tarballs.
cubic-dev-ai[bot]

This comment was marked as resolved.

cubic-dev-ai[bot]

This comment was marked as resolved.

@henrybear327

Copy link
Copy Markdown
Collaborator Author

Depending on #242 so the socket opening and closing can be functioning correctly.

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the OCI pipeline and the C runtime boundary. Three findings are inline below. One more is worth flagging but sits in pre-existing code this PR does not modify, so it cannot be anchored to the diff:

src/syscall/net-absock.c abstract-socket bind slot reservation (pre-existing, adjacent to this PR's changes): absock_register_locked populates a table slot but leaves active=false, and absock_lock is dropped before the actual bind(); the slot is only marked live later in absock_bind_commit. A concurrent guest bind() scans for !active, reuses the same slot, and overwrites its guest_fd/name/fs_path, and the duplicate-name check (absock_lookup_locked, active-only) misses the in-flight name. Two guest fds then alias one slot, so unregister leaks one socket file and its reverse-map fails. Reserving the slot under the lock in register_locked (and clearing it in absock_bind_rollback) closes the window. Since this PR reworks the same file, it may be a good time to fix it.

Comment thread cmd/elfuse-oci/gc.go
// first so index.json never keeps referencing a manifest whose blobs this
// sweep reclaims; a dry run skips that but, rooting here at pins too, still
// reports exactly the blobs a real run would reclaim.
func (s *store) liveBlobDigests() (map[string]bool, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blob GC can delete blobs a live run still needs. liveBlobDigests roots reachability only at current refs.json pins. resolveImageForUse drops the store lock once it holds the per-digest rootfs run-lock, so a concurrent pull can repin the ref to a new digest; a following prune/rmi blob sweep then removes the old digest's manifest/config/layers while a resolved-but-not-yet-unpacked run is still reading them. The cache sweep already skips busy caches via rootfsCacheBusy (gc.go:367), but this blob reachability pass does not. Also root liveness at digests whose rootfs run-lock is busy, the same way the cache sweep does, and add a test that combines a repull with prune while a run-lock is held.

Comment thread cmd/elfuse-oci/unpack.go
// Ownership is not applied here: elfuse runs as the host user and overrides
// identity at runtime via --user, so the rootfs carries only mode bits.
func unpackImage(img v1.Image, dest string) error {
if _, statErr := os.Lstat(dest); statErr == nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A symlink can redirect extraction (and execution) out of the store. os.Lstat(dest) treats any pre-existing entry as a mergeable rootfs and unpackInto then calls os.OpenRoot(dest), which follows a symlinked dest; the run path's os.Stat(rf.rootfs) (commands.go:250) follows it too. A symlink planted at the digest-keyed cache path <store>/rootfs/sha256/<hex> redirects extraction writes, or run --plain-rootfs execution, to an arbitrary directory. For store-managed paths, reject a dest that is not a real directory (i.e. a symlink) before OpenRoot and the cache-exists check; keep symlink-following only for an explicit --rootfs if that is intended.

Comment thread src/syscall/net-absock.c Outdated
# plus the end-to-end OCI run and image-lifecycle checks
# oci-conformance : OCI image-layout conformance + cross-tool interop
# (crane/skopeo/umoci) on Linux
# oci-macos : elfuse-oci darwin build, unit tests, sparsebundle

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oci-macos is ambiguous, as OCI instances built on macOS could also be abbreviated as oci-macos. Clarify.

Guest paths under /tmp, /var/tmp, and ~/.ccache are force-redirected
into the sysroot when a file is created there, so a guest's temp files
avoid collisions on the host's case-insensitive /tmp. Lookups were not
redirected: an absolute guest path absent from the sysroot falls back to
the literal host path, so a /tmp entry present on the host but not in
the sysroot was visible to stat and open.

The two disagreed. unlink, rmdir, and rename translate through the
create resolver and so addressed the sysroot, where the entry does not
exist, while stat and open addressed the host, where it does. A guest
could stat, open, and read such an entry but got ENOENT on every attempt
to remove or rename it: rm -rf of a host-visible /tmp directory failed
midway with "No such file or directory" despite -f, because readdir
listed a child that the following unlink could not resolve.

Give both resolvers one predicate for the temp roots, so the rule cannot
drift between them again, and match the roots as directories as well as
prefixes of their children. Matching only the children leaves the same
disagreement one level up, where a listing of /tmp enumerates host
entries whose names no longer resolve. ccache matches by path component
because HOME may be a macOS home directory that no other rule covers.

A host entry the guest never created under those roots is no longer
visible, which is what the create-side redirect already implied. A guest
program or interpreter stored there can no longer be loaded either,
since the ELF path resolves the same way; pass one from anywhere else.
The host fallback is unchanged for every other path, so guests still
reach host resources outside these temp roots.

test-sysroot-tmp-remove pins the regression. It stages a directory under
the runner's own /tmp, so the guest reaches it by the redirect, and
asserts that whatever the guest can stat there it can also unlink,
rmdir, and rename, that every name a /tmp listing reports resolves, and
that a guest-created /tmp file reads back. The three consistency lanes
fail with ENOENT without the change, and the listing lane fails when
only the children are redirected. The harness then confirms from the
host side that the staging directory is untouched and that the guest's
own write landed in the sysroot.

test-sysroot-create-paths asserted the absence of a TOCTOU escape by
stat'ing a host /tmp path from inside the guest, which this change makes
impossible to observe. That check moves to the harness, which runs on
the host and can still see it.
A Linux guest names files by exact bytes; the default APFS volume
matches names case- and normalization-blind, so "Foo" and "foo" cannot
coexist in one directory. Introduce the representation that lets them:
a name the volume cannot store as itself is stored escaped, and the
escape is a pure function of the name, so nothing has to be persisted
in order to reverse it.

Escaping keys on the name alone: any uppercase ASCII letter, any byte
above 0x7F, or a name already shaped like an escape. That is
deliberately conservative, because the volume's matching is far more
aggressive than case plus NFD (sharp s folds onto "ss", final sigma
folds by position, compatibility mappings apply) and nothing short of
full Unicode tables predicts it. What the rule leaves literal is
lowercase ASCII, a fixed point of every transformation the volume
applies, so two literal names cannot collide however the folding table
changes. Keying on the name rather than on directory contents also
means colliding creates never contend.

The payload has two tiers because the per-name limit is 255 UTF-16
code units, not bytes: hex up to a 125-byte guest name (readable back
with xxd -r -p), and above that twelve bits per symbol from 4096 CJK
Unified Ideographs at U+4E00, a block that neither normalizes nor
folds. A _Static_assert holds both tiers to the unit limit.

The codec is a leaf translation unit so the unit test links exactly
the code under test; probe-volume-naming regenerates the measurements
docs/filenames.md tabulates. tests/casefold-vectors.h freezes the
on-disk spellings byte-for-byte in both directions, making a format
change a deliberate migration rather than a silent test update.
Resolve a guest path to its host spelling one component at a time,
applying the escape where a name cannot be stored as itself. This is
the half of the model that has to touch the filesystem, kept apart
from the codec so the codec stays a leaf translation unit.

Each component is decided by asking the volume for the name as stored
and comparing bytes, the only way to tell "exists as spelled" from
"exists under a spelling that folded onto it": a plain stat reports
success for both, while Linux resolution is byte-exact and owes ENOENT
for the second. The two answers stay distinct all the way out: folded
is not absent, because a caller deciding whether the sysroot has a
claim on the path needs the difference. The whole path is probed
prefix by prefix, since the volume validates only the last component
and a wrong-case parent folds away silently.

The walk is seeded from a descriptor and a prefix, so one
implementation serves an absolute path measured from the sysroot and a
relative one measured from a dirfd. Along the main path nothing is
opened (each probe is a getattrlistat), so the walk holds no
descriptors and has no cleanup path. A volume that cannot report a
stored spelling falls back to reading the directory, finished with an
fstatat because a name missing from the listing may still be present
under a spelling that folded onto it.

The unit test drives staged literal names, staged escapes, symlinks,
and a dangling link; its length arm grows a path a component at a time
and requires a clean ENAMETOOLONG boundary, because a host path is
roughly twice its guest path while macOS allows a quarter the length
Linux does, and a truncated path names a different file.
Guest paths were resolved twice: proc_resolve_sysroot_path_flags
concatenated the sysroot prefix and probed existence, then
path_translate_at ran a second, independent walk that read the sidecar
index and could override the first answer. The two disagreed about when
a path falls through to the host. Fold them into one: the case-exact
walk decides each component's stored spelling and reports whether the
path resolved, so its verdict is the existence answer and its output is
the host path. A relative name resolves through the same walk seeded
from the descriptor it is measured from, and the openat2
RESOLVE_NO_SYMLINKS and RESOLVE_NO_XDEV walkers, which spelled
components the guest's way and missed every escape, resolve the same
way.

Absence is not one answer but three, and only the first may look at the
host. A path nothing claims falls through. A path whose slot is held by
a differently-spelled sibling is absent to a byte-exact reader, but the
sysroot holds something there, so the caller's own syscall reports
ENOENT; treating it as unclaimed would send a wrong-case lookup out to
an unrelated host file. A path stopped at a component that is not a
directory is the same: resolution fails there (path_resolution(7)), so
the walk records it and the sysroot spelling is returned, or "file/tail"
would be answered by whatever host path shares those bytes, ENOENT where
Linux owes ENOTDIR.

Creates ask where an absent leaf would go, so the containment flag
ladder tests create before nofollow (a renameat destination carries
both), and the walk reports the parent separately from the leaf,
replacing an access(2) probe that a folding volume answered wrongly in
both directions. Directory entries and the cwd decode on the way back
out, so getcwd never leaks an on-disk spelling.

With the mapping derived from the name, the sidecar has nothing left to
do. Its five private syscall handlers return to the ordinary
translation, and stop bypassing the /proc, FUSE, and /dev/shm handling;
the index, its parsers, the process mutex, the fcntl lock, the rollback
snapshots, and both caches are removed. Two elfuse processes sharing a
sysroot need no coordination, because there is no shared mutable state
left. The internals module table and the usage notes described that
machinery, so both are rewritten rather than left naming a deleted file;
the notes also record that a sysroot written by the old encoding has to
be recreated, the one guest-visible consequence of this commit.

Smaller corrections ride on the single walk: a trailing separator keeps
Linux's ENOTDIR by asserting directory-ness on the host path; a sysroot
mounted at "/" no longer turns a leaf's parent into an empty string; and
a path unlinked by a peer between probe and realpath reports ENOENT
rather than a containment veto's ELOOP, since neither the path nor a
vanished sysroot leaves anything reachable once it stops resolving.

The contract lands in two lanes, name-unique and name-relative, which
also run against the qemu reference kernel, where a real Linux kernel
measures what they assert. Neither can run in the elfuse lane, which has
no sysroot, so the matrix gains ELFUSE_SKIP: the inverse of QEMU_SKIP,
with one membership helper and a check-format gate rejecting a label
that names no registered test or sits in both lists.
The naming design rests on claims a green build does not exercise;
give each its own lane.

Edge shapes: test-sysroot-root drives the degenerate one-character
prefix over the read-only macOS root; test-nosysroot-literal-names
pins that without a sysroot nothing decodes and an escape-shaped host
file means itself; test-sysroot-chdir requires getcwd and
/proc/self/cwd to report spellings the guest can hand back to chdir.

Names: test-sysroot-name-i18n pins the pairs the volume considers
equal that the guest must see as two files (sharp s, positional
sigma, ligatures, normalization families, no-case scripts, invalid
UTF-8), and in csapfs mode pins the two documented divergences of
case-sensitive APFS instead of skipping. test-sysroot-name-length
pins that a guest keeps all 255 bytes Linux allows across both tier
boundaries of the escape, and that 256 is refused.

Lengths are two separate budgets, and the second one is the host's:
test-sysroot-pathmax pins ENAMETOOLONG where macOS's 1024-byte
PATH_MAX undercuts the guest's 4096, since a host path runs to roughly
twice its guest path and a truncated one names a different file. It
stays out of the qemu matrix because a real Linux kernel has no such
ceiling and correctly builds every path the lane expects refused.

Reading back what an older build wrote: test-sysroot-corpus stages a
tree of on-disk spellings byte-for-byte from tests/casefold-vectors.h
and opens each strictly by guest name, which is the direction an
existing sysroot exercises after an upgrade. Staging happens on the
host, so nothing it asserts derives from the codec under test.

Host-staged names: a sysroot is an ordinary directory anything may
write into, so test-sysroot-name-staged fixes the meaning of names
elfuse never produces: a well-formed escape decodes no matter who
wrote it, anything merely resembling one means itself, and when both
spellings of one name are staged the literal wins the lookup while
the listing reports both.

Concurrency: nothing serializes name creation, because the on-disk
spelling is a function of the name alone, the load-bearing claim.
test-sysroot-name-race forks children that create colliding members
and race O_EXCL for a single kernel-picked winner, and runs in check
ten times over, because one round is cheap and a single round can
miss the window. test-sysroot-name-soak churns creates, renames,
unlinks, and listing scans from threads and forked children against a
deadline; it is registered as a manual target and kept out of check
for its runtime. Neither proves the absence of a race, and both
headers say so.

The name lanes also run against the qemu reference kernel, where a
real kernel confirms Linux keeps each pair apart, and are skipped in
the elfuse lane for want of a sysroot.
The representation half of docs/filenames.md landed with the codec.
Fill in the resolution half: how a path is resolved a component at a
time, what each answer from the volume means, and the two properties
that fall out of deciding a name's spelling from the name alone:
that a guest name is reachable through exactly one on-disk entry, and
that nothing in the subsystem takes a lock, since each operation is
one atomic syscall and no second object records what a name means.

A section on symlink targets records the one thing the model cannot
represent: a link stores the bytes the guest gave it, because readlink
has to return them, so a target naming something stored escaped cannot
be followed by the host. The pure-function claim is narrowed to what
is true: two rows of the resolution table do consult the directory,
both only for a fold-stable name whose escape an outsider staged.

The dynamic-linker walkthrough still described a sysroot-first openat
rather than the one forward resolver every path-taking handler now
uses, so it is rewritten to point at path_translate_at, and its
limitations section now names this document instead of claiming
nothing is tracked: what the sysroot volume cannot represent is a
real limitation of that path, recorded here.
A symlink records the bytes the guest wrote, and it has to: readlink(2)
owes those bytes back, so a target cannot be translated on the way in.
But those bytes name a guest path while the disk holds host spellings,
so handing them to the host kernel looks somewhere else: a guest could
not follow its own symlink when a target component was stored escaped,
and an absolute target resolved from the host root instead of the
sysroot.

Follow targets in the guest namespace instead. The walk stops at a link
it must pass through and reports where; the resolver reads the target,
joins a relative one to the link's directory or lets an absolute one
replace the path, appends the remainder, and resolves that as an
ordinary guest path. The loop is iterative, since a chain may run to
MAXSYMLINKS and each step needs a whole path buffer. Detecting the link
is free: ATTR_CMN_OBJTYPE rides along on the probe the walk already
makes.

POSIX fixes which components are followed: every intermediate one, and
the last only without nofollow. All four resolvers learn the rule:
lookup, create, the descriptor-relative translation, and the
RESOLVE_NO_SYMLINKS precheck, for which a stopped-at-a-link verdict is
the ELOOP it exists to report. The precheck asks the walk directly,
because splicing a link into its target hides it from any later scan;
the descriptor-relative translation reuses the host path its containment
check already resolved, keeping one flag mapping rather than a second
copy that can drift.

An absolute target resolves against the sysroot but does not inherit the
host fallback a typed path gets: anything able to write a symlink into
the tree could otherwise hand the guest a file from outside it. The
splice shares the existing truncation-checked concatenation, and
MAXSYMLINKS moves to path.h so the path layer and the resolvers cannot
disagree about when a chain has run too long.
A watch is backed by kqueue on a host fd, but the path it is added
under is a guest path: opened raw, an absolute watch lands in the
host's namespace and reports ENOENT for a directory the guest can
chdir into. Route it through path_translate_at like every other
path-taking syscall, refusing FUSE and synthetic /proc objects with
ENOSYS since kqueue cannot observe them.

Those two are the whole refusal. The open path's
path_might_use_open_intercept is not reused for it: that predicate is
a prefilter, true for every name beginning "/dev" (four bytes, so
"/development" too), for the sysfs CPU tree, and for /etc/passwd
whenever the sysroot carries no copy. Each of those has a real host
vnode behind it (a /dev/shm leaf is redirected to one by the
translation itself), and Linux grants a watch on all of them, so
gating on it would answer ENOSYS where a watch is owed. A lane pins
the /dev/shm leaf against that.

That redirect makes the leaf a real host file, so the watch is opened
nofollow. A guest may write a symlink into the /dev/shm backing
directory, and following one would report the existence of, and every
change to, whatever it names, including a host path
is_guest_system_path() exists to keep the guest from addressing at
all. Linux follows here, but the redirect is elfuse's own and every
other consumer of a shm leaf already departs the same way. A second
lane pins that beside the one granting the watch.

The snapshots feeding named IN_CREATE/IN_DELETE events read raw
directory entries, so on a folding sysroot an event named the stored
.ef= spelling, bytes the guest never wrote and cannot stat. Decode
each entry through path_translate_dirent_name, the choke point
getdents64 uses; an over-long name is skipped exactly as getdents64
skips it, and any other failure keeps the previous baseline rather
than diffing every decoded child as deleted. The decode helper drops
a dirfd parameter it never used (decoding depends on nothing but
the name), so this caller, which holds no guest descriptor, does not
have to invent one.

The lane's recipe asserts host-side that the fixture survives only
under its escape, so a passing guest lane proves the events were
decoded. Events are collected by reading a nonblocking fd: the
emulation pumps its queue on read, and poll-only readiness is a
pre-existing gap this change does not touch.
execveat with a dirfd-relative name opened the raw guest bytes: under
a casefold sysroot a guest-created binary exists on disk only under
its escaped spelling, so the open reported ENOENT for a file execve
runs fine, and could have resolved a case-colliding host-literal
file instead. Translate the name like every other *at handler,
refusing FUSE and synthetic /proc objects, and open the translated
spelling with O_CLOEXEC so a concurrent execve on another vCPU thread
cannot inherit the descriptor.

The identity that resolution publishes needs the reverse map: left as
the resolved host path, /proc/self/exe reported the sysroot prefix
and .ef= spellings, bytes the guest never wrote and cannot exec,
which is how a self-re-exec stops working. sys_execve derives the
guest-visible identity with path_host_to_guest while keeping the host
path for the actual open; proc_readlink_self_exe replaces its bare
prefix strip with the same reverse map; and /proc/self/fd/N runs its
F_GETPATH result through it too.

The lane's recipe asserts host-side that the staged binary sits on
disk only under its escape, so the passing exec lanes prove
resolution actually crossed the boundary.
The interpreter resolver kept its own three-strategy probe from the core
loader: a literal sysroot concatenation checked with access(2), then
/lib/<basename>, then the guest bytes. The concat probe answers with the
volume's folding lookup, so a wrong-case interpreter spelling was
accepted where Linux resolution is byte-exact, and a hit skipped the
containment, /dev/shm, and FUSE handling every other exec path gets.
Route the decision through the ordinary translation: a usable translated
path wins, and the /lib/<basename> fallback still catches store-style
interpreter paths. elf_resolve_interp stays with the core bootstrap,
which cannot call into the syscall translator.

Usability is judged under the rule the open then uses. A shm redirect is
opened O_NOFOLLOW, so its leaf is probed without following: access(2)
follows, and a symlink in the shm backing directory would answer usable
for a path the open refuses with ELOOP, skipping the fallback that would
have found the loader. Everywhere else the probe still follows, since an
interpreter is routinely a symlink.

On this tree the escaped-spelling case already reached the translator
through the old code's literal fallthrough, so the two lanes are
regression guards rather than observed-red fixes: one pins the /lib
fallback for an absent store path, the other an interpreter staged under
an escaped directory and exec'd from inside the guest. Both skip when
the musl fixtures are absent.
A pathname socket's address is a filesystem path, but every sockaddr
crossed the boundary through a raw byte copy: bind created the socket
file at the host-literal path (outside the sysroot for any guest
path not mirrored there), and connect, sendto, and sendmsg aimed at
the same wrong namespace, so two guest processes could not rendezvous
through a socket in a guest-created directory. Route pathname
addresses through path_translate_at: bind uses create semantics, so
colliding socket names coexist and bind, stat, connect, and unlink
agree on which file a name means, with EADDRINUSE for an occupied
name. getsockname, getpeername, accept, recvfrom, and recvmsg decode
returned addresses through path_host_to_guest, so the guest reads
back the spelling it bound.

A translated host path routinely overflows the 104-byte macOS
sun_path (the sysroot prefix plus an escape more than doubling a
component), so over-long paths divert through a short symlink in the
private absock namespace dir; bind through a dangling symlink creates
the socket at the target and connect follows it (probed on macOS 15).

Two details of that readback come with it. recvmsg reported the macOS
address length beside the translated bytes it wrote; the two were
interchangeable until translation made them differ, and a guest sizing
sun_path from msg_namelen then read past the address into its own
buffer. And the namespace dir is recognized from the namespace id
rather than from having created it, so a forked child (a fresh
process that inherits the id) undoes the shortening symlink too,
matched on a whole path component so one namespace cannot claim
another's by prefix.

A /dev/shm leaf carries the never-follow rule that redirect exists
for, and bind and connect take a sockaddr rather than a dirfd and
at_flags, so the rule cannot ride on an open flag here and is checked
outright. Following a guest-planted link there would bind the socket
at the link's target, outside the tree, and would answer connect with
ENOTSOCK for a host file that exists against ENOENT for one that does
not, telling the guest whether any path exists. Both reach what
is_guest_system_path() denies the guest by name.

The exit sweep for that shared namespace dir arms in
absock_ensure_dir_locked, the one point where on-disk state first
appears, rather than in sys_bind's abstract-socket branch alone. The
sweep learns the shared exit orders: each process unlinks its own
table-tracked sockets, only the namespace's creator retires untracked
symlinks, any participant may rmdir, and only symlinks are swept:
the abstract-socket backing files in the same directory belong to
whichever process bound them, including children that outlive the
owner. The names lane asserts host-side that a socket survives only
under its escape; the lifecycle lane covers both fork exit orders.

With socket addresses decoded, every surface that hands names back to
the guest now participates; docs/filenames.md names the full decode
boundary in one place.
Two lanes join the suite, both of them oracles rather than
expectations. check-name-caseexact re-runs the name suite on a
case-sensitive APFS sparsebundle, where the volume itself enforces the
byte-exact matching the tests assert, so an expectation that fails
there disagrees with a real Linux filesystem whatever the folding lane
says of it; this lane is what surfaced the ENOTDIR and vanished-path
fixes in the resolver.

test-sysroot-path-matrix closes the class the hand-written tests
sampled: every late bug on this branch sat in one cell of a cross
product (addressing mode x operation x path shape x name class)
that no one had thought to visit. The matrix enumerates the product
and holds each cell to path_resolution(7)'s oracle: an absolute, a
cwd-relative, and a dirfd-relative spelling of one file must agree on
result, errno, and object, with creates verified through the
canonical absolute spelling. On first run it caught an absolute
openat2(RESOLVE_NO_SYMLINKS) missing an in-sysroot intermediate link
and a relative rename below an escaped directory failing where the
absolute spelling succeeds. It runs on the folding tmpdir, again on
the case-sensitive volume, and against the qemu reference kernel,
where agreement is a measurement rather than an assertion.

Putting the matrix on the byte-exact volume needs the oracle's closing
sweep narrowed. That sweep fails the lane if any entry is stored
escaped, which is the right invariant for the name tests but not for
this one: an escape-shaped literal is one of the matrix's name
classes, so a byte-exact volume owes that name back unchanged and the
sweep read correct behavior as a violation. It now prunes the matrix
subtree and checks that subtree positively instead (a name needing
an escape on a folding volume has to be stored literally here), which
keeps the coverage the prune would otherwise drop.

The testing guide describes the lanes by family and states the shared
recipe contract (a self-provisioned sysroot with a host-side
on-disk assertion after the guest exits) instead of enumerating
targets that drift.
elfuse_launch owns guest bring-up: guest_bootstrap_prepare, the
FUSE-temp unlink, the sysroot casefold probe,
vCPU creation, GDB init/sync/wait, the run loop, gdb_stub_shutdown,
the shim counter and syscall histogram dumps, and guest_destroy.
main() retains the original CLI argv (proctitle rewriting), option
parsing, sysroot provisioning, the shebang loop, the --gdb x86_64
guard, host cwd, and the heap resource cleanup, and now hands off
through launch_args_t so other launchers (the OCI run helper) can
share one bring-up path.

The launch_args_t envp field generalizes the old hard-coded environ:
NULL keeps the host environment, so main()'s behavior is unchanged.
Bring-up failures unwind through a single fail label instead of
repeating the guest_destroy-plus-unlink tail at every error site.

Ownership of the FUSE-materialized temp ELF moves with the bring-up:
elfuse_launch owns the unlink from the prepare call onward (teardown
and the post-prepare error paths), and main() drops its claim before
handing off, so main's shared goto unwind cannot double-unlink a path
whose ownership has been transferred.

The embedded shim blob include moves along with its only consumer, so
shim_bin has a single object definition site.
An OCI image front end needs to set the guest identity, working
directory, and environment without patching the runtime; the new
flags map onto launch_args_t fields and `elfuse-oci run` drives
exactly this interface.

The --user identity is staged before bring-up (proc_set_initial_ids)
so the auxv AT_UID/AT_GID snapshot taken by build_linux_stack matches
what getuid()/getgid() later report. --workdir rejects non-absolute
paths up front instead of silently resolving them against the host
cwd, and is applied by elfuse_launch after the casefold probe so the
translation sees the sysroot's real case behavior. --env/--clear-env
build the guest environment with env(1) semantics: KEY=VAL sets,
bare KEY inherits from the host environ, --clear-env starts empty;
with neither flag given envp stays NULL and the host environ is used
unchanged.

The new heap resources join main()'s shared goto unwind: envp,
workdir, and the raw --env override array are released at the single
cleanup label on every exit path.
elfuse-oci is a standalone Go binary that owns the OCI image pipeline;
elfuse itself stays a pure Linux syscall-to-Darwin runtime with no OCI
commands. This first slice is the acquisition half: an OCI image-layout
store plus the pull and inspect commands, built on go-containerregistry
($ELFUSE_OCI_STORE or ~/.local/share/elfuse/oci by default).

The store is a spec-shape image layout other tools can read, with a
refs.json pin table mapping references to manifest digests. An exclusive
flock serializes refs.json/index.json updates so concurrent pulls cannot
lose pins (the cold-store bootstrap of oci-layout and index.json runs
under the same lock, double-checked so a warm store skips it), pin
persistence syncs the temp file and directory around the rename, and a
nil-object refs.json is rejected as corrupt instead of treated as empty.
addImage distinguishes genuinely-absent from unreadable descriptors by
scanning index membership, so store corruption surfaces rather than
duplicating entries. digestFor returns a distinct errNotPulled for a
missing ref so later callers can tell "not pulled" from "store broken".

Two helpers keep the plumbing in one place: every subcommand opens the
store through commonFlags.openResolvedStore (resolve the store path,
then open the layout), and store.withLock scopes lock-held sections; pin
and addImage wrap their load-modify-save cycles in it so a critical
section cannot leak its lock on an error path.

pull resolves the requested platform (default linux/arm64) and validates
--platform shape up front; inspect prints the manifest and config
summary (or --json) and propagates digest/size and writer errors instead
of reporting partial output as success.

Credentials come from the ambient default keychain, but its resolution
is time-bounded: it shells out to whatever helper the Docker config
names, and go-containerregistry drops the context around that exec, so a
wedged helper would otherwise hang the pull with no output. A wrapper
keychain caps the wait and fails with an explanation and the
DOCKER_CONFIG escape hatch instead; a progress line is printed before
the pull so it is never silent.

Tests cover the pin table, store locking, error kinds, flag parsing, and
the command dispatch, with cranePull as a swappable seam so no test
touches the network.
Apply a stored image's layers into a rootfs directory, whiteouts and
all, so `unpack` (and later `run`) can materialize a filesystem tree
from the store without any external tool.

Layer application is hardened against hostile archives: extraction is
bounded by os.Root so no entry, symlink, or hardlink escapes the
destination; member names and hard-link targets archived absolute
(GNU tar -P builders) are applied root-relative, as other OCI
consumers do; parent components are Lstat'd and a symlinked or non-dir
intermediate is replaced with a real directory (containerd/Docker
behavior); opaque whiteouts clear through a real directory only, are
order-independent within a layer, and an invalid bare .wh. entry
fails extraction instead of deleting its parent; a directory entry
replaces a lower-layer non-directory. File bodies are bounded by the
tar header size and short bodies are an error, and permissions are
finalized with an explicit chmod so the host umask cannot skew modes.

setuid/setgid/sticky bits are re-applied where the host allows it, and
degrade gracefully where it does not. An unprivileged chmod that sets
setuid/setgid is rejected with EPERM on macOS when the unpacked file's
inherited group is one the invoking user is not in (a new file takes
its parent directory's group under BSD semantics, e.g. wheel under
/tmp, not the tar's root/shadow owner). Since the rootfs is owned by
the invoking user those bits could not be honored at runtime there
anyway, so they are dropped with a warning naming the lost bit rather
than aborting the whole unpack, which is what lets Debian-family
images and their shadow suite (chage, passwd, ...) unpack at all.
Reported at
sysprog21#191 (comment)

Unpack stages into a temp sibling directory and renames into the
final cache path (keyed by manifest digest under <store>/rootfs/), so
a concurrent reader never observes a partial tree and the loser of a
rename race adopts the winner's complete one. A failed unpack removes
only what it created.

Tests drive whiteouts, opaque ordering, parent-symlink replacement,
hardlink identity, exact-size reads, mode preservation, the special-
bit degrade decision, and the staged-rename semantics over synthetic
layer tarballs.
run is the last pipeline stage: resolve the image config into a concrete
runspec, materialize the rootfs, and exec the existing `elfuse --sysroot
<rootfs> ...` positional launch path, reusing elfuse's HVF bring-up,
shebang, and dynamic-linker plumbing rather than reinventing guest
launch. elfuse is located as a sibling binary ($ELFUSE_BIN overrides for
tests) and replaced via exec so the shell reaps the same pid and
terminal signals reach the guest directly.

The runspec resolves Entrypoint/Cmd/Env/User/WorkingDir with the usual
precedence (--entrypoint drops image Cmd; --env and --clear-env follow
env(1) semantics; --workdir must be guest-absolute). A PATH is
guaranteed: when neither the image config nor --env supplies one,
Docker's conventional default is appended, so a guest whose image omits
PATH still has a search path after the --clear-env launch. A relative
path command resolves against the working directory and a bare name
against the merged PATH inside the image rootfs, following Docker's
exec-form rules (elfuse resolves the initial ELF before applying
--workdir and does no PATH lookup, so the launcher must); a config-only
image WorkingDir no layer ships is created at run time, as runc does. A
symbolic --user or image User is resolved against the image's own
/etc/passwd and /etc/group through os.Root-bounded, no-follow opens, so
a crafted rootfs cannot redirect resolution to host account files.

run auto-pulls only when the ref is genuinely absent (errNotPulled) and
surfaces store corruption instead of masking it behind a network pull;
an explicit --platform must match the pinned image so a ref pulled for
another architecture is not silently launched. Before exec, host-truth
/etc/{resolv.conf,hosts,hostname} are injected into the rootfs through
the same os.Root bounds.

Tests cover runspec resolution and precedence, symbolic user lookup,
runtime-file injection, and the exec argv shape via the execElfuseForRun
seam and a subprocess exec probe.
A plain directory rootfs on the default case-insensitive APFS volume
folds Linux filenames that differ only by case. Default runs now use
a case-sensitive APFS sparsebundle per pinned manifest digest, with a
per-run clonefile COW rootfs so guest writes never mutate the warm
base tree and repeated runs skip the unpack.

Liveness and lifecycle transitions are decided by per-digest advisory
flocks in the bundle directory (bundlelock.go), not by pids or
directory scans: every live run holds run.lock shared from before the
volume is attached until guest exit, so a killed run cannot leak
liveness, and attach.lock serializes provisioning (always acquired
before run.lock, making the exclusive-to-shared downgrade in
provision race-free). Provisioning reuses a mount a live run still
holds and only force-detaches one proven stale by an exclusive
run.lock probe, so a second run of the same digest can never rip the
rootfs out from under a live guest. --keep clones carry an
.elfuse-keep marker so sweeps preserve them.

The plain-rootfs path remains available with --plain-rootfs, and the
non-Darwin stub keeps elfuse-oci buildable for
pull/inspect/unpack tests on Linux.

Darwin tests drive runCaseSensitive through seams for provisioning,
clonefile, spawn, cleanup, and exit, and cover sparsebundle
provisioning, mount/detach, attach-failure teardown, and the lock
protocol with a fake hdiutil; the mount-probe and force-detach hooks
are function variables so tests need no real disk images.

The spawn path intercepts SIGHUP alongside INT/TERM/QUIT and installs
the handler before the child starts, so a hangup or an early signal
still flows through the forward/reap/teardown path. elfuse is invoked
with a "--" separator before the guest command, so an image
Entrypoint beginning with "-" cannot steer the host launcher. hdiutil
attach failures keep stderr in the error message (stdout stays clean
for the plist parse), and the parsed mount path is XML-entity-decoded
so a store path containing "&" or quotes still round-trips.
Add list/images, rmi, and prune on top of the OCI image-layout store.
rmi and prune use reachability GC: shared manifests, configs, and layers
stay on disk while any remaining ref reaches them.

Cache cleanup handles plain rootfs caches and macOS sparsebundle caches
through platform-specific seams, with safety rules for active workloads:
a prune without --all never touches a still-pinned digest's cache, and a
cache a live run still uses is never reclaimed. Liveness is the same
flock discipline on both cache kinds: the bundle's run.lock, and for the
plain digest-keyed rootfs a sibling <hex>.lock that a run holds shared
from before the existence probe until guest exit. The plain run path
execs elfuse in place, so the lock descriptor is made exec-survivable:
the kernel releases it exactly when the elfuse process exits, SIGKILL
included. prune takes each lock exclusively non-blocking and skips busy
caches (dry runs never advertise them); rmi refuses a busy cache even
with --force, before any pin or descriptor is touched. The lock is a
sibling of the cache dir, not inside it, because the dir is the guest's
/ and its existence is unpackImage's atomic-rename publication signal.

prune's GC sweep and list's snapshot run under the store lock, closing
the windows where a concurrent pull's fresh blobs could be reclaimed
before their descriptor lands or a listing could observe a half-removed
ref. prune scopes its GC-plus-cache sweep with store.withLock and drops
the lock before the reporting output; list and rmi keep the explicit
acquire-and-defer because their entire body is the critical section, so
a closure would only add indentation. list reports the full
os/arch/variant platform.

Two corrections to the unpack path come with the locking, because both
are what the lock discipline needs to be true. unpackImage now takes the
caller's already-resolved image rather than re-resolving a ref: the
caller keys the cache by the digest it resolved, so a re-resolution
could observe a different pin from a concurrent repull and fill digest
A's cache with image B's content. resolveImageForUse pairs the
resolution with the digest lock, which is why the fix lands here rather
than in the commit that introduced unpackImage. Layer application also
tracks the ancestors of each entry, so an opaque whiteout arriving after
an implicit parent directory no longer clears content the same layer
just added.

Tests cover reachability GC (shared blobs, stale temp blobs, digest
prefixes), the prune and rmi busy semantics for both cache kinds (live
holder, stale lock file, dry-run visibility), lock survival across the
exec boundary (FD_CLOEXEC clearing plus a child-process flock-lifetime
proxy), which runs hold the lock at exec time, and end-to-end command
wrappers over pull/run/list/rmi/prune. The darwin cache sweep is driven
through the isMountPointFn and detachForce seams with
reapSweepableClones deciding the reap set, so the sweep is testable
without real disk images.
The on-disk store is the contract: pulls must produce a valid OCI
image-layout that other tools can read and that agrees with registry
truth on the manifest digest. Conformance tests pin the layout shape,
and scripts/oci-interop.sh cross-checks the store with crane, skopeo,
and umoci.

CI splits by runner capability: a Linux job runs the pure-Go store
paths (pull, unpack, inspect, lifecycle) without Hypervisor.framework,
a hosted macOS job builds and tests the darwin sparsebundle code and
drives a run-less pull/inspect/list/rmi/prune lifecycle smoke, and the
self-hosted release leg boots guests end to end under HVF: the
alpine:3 default-entrypoint smoke plus a full pull -> inspect -> list
-> run -> rmi -> prune lifecycle that runs python:3.12-slim with an
--entrypoint override and asserts the teardown half of the lifecycle.

The lifecycle teardown covers all three reclamation guardrails: a
plain rmi reclaims the cold cache with the image, run --keep output
refuses rmi without --force, and a live --plain-rootfs guest parked
in sleep pins its cache; prune --cache --all must skip it and rmi
must refuse until the guest exits, the only end-to-end exercise of
the run-lock descriptor riding through the exec into elfuse.
Cover the two-binary model in README and docs: usage.md documents the
elfuse-oci commands and flags, testing.md the offline/CI
validation split, internals.md the host-literal path fallback
semantics, and oci-design.md records the design rationale (the C/Go
boundary, the image-layout store with its refs.json pin table, layer
application, run paths, and lifecycle GC), plus an explicit
scope-and-limitations accounting of which OCI features are and are
not implemented.

oci-design.md also records the concurrency model: store metadata is
lock-serialized, per-digest sparsebundle state is coordinated by the
attach.lock/run.lock pair, and the plain digest-keyed rootfs cache is
guarded by a sibling per-digest lock a run holds across the exec into
elfuse, so prune skips and rmi refuses a cache a live guest still
uses. usage.md notes the resolv.conf fallback nameserver and its
split-DNS implication.

README states the positioning up front: OCI images are consumed as a
distribution vehicle for Linux root filesystems, replacing hand-built
--sysroot trees, and the non-isolation limitations (host-path
fallback, shared network identity, PID space, and clock) are called
out explicitly rather than implied.
Issue sysprog21#224 profiled five real images (python:3.12-slim, node:22-alpine,
golang:1.23-alpine, eclipse-temurin:21, gcc:14). Add one CI job per
image that boots the image under HVF via `elfuse-oci run` and drives
that image's operations, so a change that breaks any of them is caught
on the PR rather than by hand.

A shared driver, scripts/ci/oci-workload.sh <key>, maps each key to its
image and guest workload under scripts/ci/workloads/ and asserts a
per-image sentinel token:

- python: a single-threaded SQLite insert plus an aggregate query, a
  small file write/read/checksum fan-out, and a JSON round-trip.
- node: in-guest compute (fs/crypto/zlib/JSON) plus an HTTP server the
  job reaches over the host loopback; elfuse maps guest sockets to host
  sockets and does no network-namespace isolation, so a guest bound to
  127.0.0.1 is reachable host-side.
- go: gofmt over a tree of misformatted fixtures the workload writes
  itself, asserting the file set it names, the bytes it produces, and
  that a second pass names nothing. The toolchain binaries are Go
  programs, so this drives the runtime's own scheduler and raw syscalls;
  it compiles nothing, because the guest compiler dies on SIGHUP before
  finishing a package.
- jvm: javac + java exercising collections, file I/O, SHA-256, an
  8-thread pool, and a subprocess.
- c: a small multi-file make project plus a heavier single translation
  unit compiled with gcc -O1.

The go and python lanes stay inside the runtime's current limits;
heavier variants that stress those limits, including an in-guest Go
build, are submitted separately.

The jobs run only on self-hosted Apple Silicon because `run` needs
Hypervisor.framework. They share a composite action that fetches the
elfuse binary from build-macos and builds elfuse-oci, and each keeps a
warm per-key store on the runner's persistent disk so only the first run
pulls over the network. gcc:14 and eclipse-temurin:21 ship the shadow
suite, so those jobs also exercise the unpack setuid/setgid degrade end
to end.

The existing python workload moves under scripts/ci/workloads/;
oci-lifecycle.sh follows the new path.
The run smoke proves an image boots and computes; these checks cross
the guest-execution seams it does not. A pathname AF_UNIX socket is
bound inside a guest-created directory with a getsockname round-trip:
the runtime translates sun_path into the sysroot on the way in and
must reverse-map it on the way out, and the sparsebundle clone's deep
host path additionally forces the over-length shortening indirection.
A cold-provision boot (blobs cloned into an ephemeral store with the
cs/ bundles dropped, so no network) must report the unpack, and the
following warm re-attach of the same digest must boot without
unpacking again. A dynamically linked from-image binary runs under an
explicit entrypoint so PT_INTERP and its shared-object closure must
resolve inside the rootfs.

Wired as a runtime-macos Release-leg step beside the run smoke,
sharing its warm store (python:3.12-slim joins alpine and debian
there).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants